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Introduction 



Multiple natrix sampling is a nsychonetric procedure for 
cstinatinp nrou]) lunraincters , It involves the simultaneous random 
sampling of l^otli itens and examinees* Although educational research 
specialists and oducators in general have lonj^ l^een aware of the 
advantar^es of cxanincc sampling, it was not until the sixties that 
research specialists beran experimenting with matrix sampling 
procedures . 

Tlio early studios helped clarify the potential of the item 
samplinr ontinn of multiple matrix sampling. Many of them, however, 
followed a rosonrcli paradigm that restricted the roneralizability of 
the findinrs. Tlic paradigm involved the extraction of multiple 



matrix snnnlinr' ontimates from an existing matrix of examinee-item 
responses collected by the administration of an entire test to a group 
of students. In direct contrast to sampling from an existing response 
matrix is tlie way in wliich matrix sampling would l^e used in an applied 
situation: cacl^ student would take only a fractional sample of the 
items on the test. Tlie differences between having a student respond to 
all 100 items on a test and having him respond to, for example, five 
items is obvious. Any of a number of error factors, such as anxiety, 
motivation, fatigue, etc., could operate to make examinees respond differently 

^A nnncr presented at tlie annual meeting of the American Educational 
Q Research Association, Chicago, IL, April 17, 1974, 
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in the two sLtua linns. 

A review ot the literature identified several studies that 
deviated from the early paradigm and that were relevant to the issues 
being invcsLignipd in this study. Owens and Stufflebeam (1967) 
administered mntrix tests to 3330 fourth grade students from both 
advantaged and d If^atlvantaged neighborhoods. Each student responded to 
a matrix lest of eitlier three, six, or 12 items. The authors concluded 
that the item sninple estimates of the mean were generally closer to the 
computed j)u()u t a I i(in value than comparable examinee sample estimates. 
Item sample estimates of the variance were not as precise as variance 
estimates of ci^nparable examinee sample estimates. Although the 
students in tlie Owens and Stufflebeam study responded to matrix tests, 
the remaining itemfj of the test being sampled, the Metropolitan Reading 
Test , were administered in conjunction with the administration of the 
matrix tests. 

Calien, Romberg^ and Zwirner collaborated on a pair of studies 
tliat invol ved il»o atlministration of matrix tests (J 970, 1973) » The 
first study ftivolvrd the use of multiple matrix sampling to estimate 
tlie per f oriii.i nr<; of nJntli grade students from 81 schools on a 50-item 
mathematics lost. The matrix sample estimates of the mean preserved 
the relative rank ordering of the schools; however, the multiple matrix 
estin»ates wore systematically higher than the actual school means. 

The second Cahen, Romberg, and Zwirner study included an in- 
teresting variation. The test sample was a 2A-'ltem Proj ect Talent 
M a thcmnt ics Test . The population of interest was twelfth grade students 
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from J3 schooJs Liiai participated in the Natioaai Longitudinal Study 
of Mathematical Ahilities. Kalf of the students in each school took 
the total test on the first day of testing and the item sample sub- 
tes^s on tho .second day of testing. The other half took only the item 
sample sul)tc»st: on the second day. The authors concluded that matrix sample 
estimates again provided reasonable estimates of the group mean and 
that taking, the total test on the first day did not affect student 
performance on li)c matrix tests on the second day of testing. 

Neither Lho Owens and Stuff lebeam study nor the two studies by 
Cahen, Romborj;, and Zwirner were specifically designed to test for 
the existence of a context effect. Sirotnik (1970) designed a study to 
test directly \'ov context effect. The Sirotnik study involved the 
direct comparison of th^: matrix sample estimates extracted from total 
test data (treatmrnit A) with matrix sample estimates computed from data 
collected by the independent administration of matrix tests (treatment 
U) • Matrix sample estimates of student performance on three different 
tests, vcn:al)ii 1 ar y , matliematics , and attitude toward reading, were 
collected under vnch treatment. A multivariate analysis of variance 
design was ut i 1 L7:ed to test for systematic differences due to context. 
None were fotmd, Sirotnik pointed out in his (iiscussion that an 
insignificant rosiilt on a single test of a null hypothesis does not 
prove i\\i\t tiio null hypothesis is true and indicated a need for the 
study to be replicated. 
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Statement of the Problem 

The purpose of this study was to test the feasibility of the a 
priori use of multiple matrix sampling procedures in a particular set- 
ting, the elementary school, with a particular type of instrument, a 
commonly used, nationally normed, commercially published achievement 
test. Specifically, the study focused on three h>7)otheses: 

1, The chanee in item context which is necessitated by the 

a priori or applied use of multiple matrix sampling does not signifi- 
cantly affect the matrix sample estimates of the population mean and 
variance. 

2, Recent previous exposure to the items being sampled does not 
significantly affect the matrix sample estimates of the population 
mean and variance. 

3, The a priori use of multiple matrix sampling procedures 
described in this study will result in estimates of the population 
mean and variance that are as accurate as the estimates obtained from 
examinee sampling procedures based on the same number of observations* 

Methodology 

The study involved 124 fourth grade students, 119 fifth grade 
students, and 109 sixth grade students who were attending tv;o different 
elementary schools. Both elementary schools were part of a consolidated 
Nebraska school district. 
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The instrument > The tests used in the study were three subtests 
of Form 5 of the Iowa Tests of Basic S];ills , The criteria for the 
selection of the subtests were that (1) each subtest be representative 
of a different content area, (2) each subtest use different item formats, 
(3) each subtest lend itself to simple sannlin^ procedures, and (4) each 
subtest contain items renroducible in black and white offset. These 
criteria eliminated most of the other subtests. The three subtests 
chosen. Vocabulary, Spellir,^, and Mathematics Concepts, were represen- 
tative of three different item formats and two distinctly different 
content areas. 

Sampling plan ^ Each of the nine subtests, three subtests at 
three pradc levels, was subdivided into six matrix tests. A stratified 
sanplinp plan was used to assij^n each item within eac]\ subtest (for each 
rrade) to one of the matrix tests. The items were stratified accordine 
to difficulty. The stratified sampling plan was used to insure that the 
matrix tests were of approximately eoual difficulty levels. The decision 
to use six matrix tests per subtest wa^ based on the need to have the matrix 
tests larf^e enoucrh so that examinees would see them as havinp substance but 
yet have the tests short enough so that the use of multiple matrix sampling 
resulted in a viable savings of time. The number of items within any in- 
dividual matrix test ran^red from six to eipht. The number of items within 
any set of matrix test consisting of a vocabulary test, a spellint^ test and 
a mathematics test varied from 18-20 for fourth graders, 21-25 for fifth 
graders and 21-24 for sixth j;^radcrs. 

The matrix tests were randomly assigned to examinees. Matrix 
tests for each subtest were assijzned independently so tliat most examinees 
were assifrned unique combinations of matrix tests. Two 
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sets of matrix tests were assigned to each participant. The first 
set was used to collect data for Ostiniates 1 and 2; the second was 
used to collect data for Hstiinates 3 and 4. 

Procedures . Three sets of data were collected, the results of 
the two administrations of the matrix tests and the administration of 

^Q^^'^ Tests -of Basic Skills battery. The three sets of data repre- 
sented four unique combinations of context and exposure. A set of nine 
nultinle matrix sample estimates, one for each of the three subtests at 
each of the three prade levels, was computed for dlta representing each 
of the four combinations of context and exposure. The following four sets 

multiple matrix sample estimates are summarized in Tahle T. 

Ksti^nntt^ 1. Data were collected by Che administration of Set 1 
of the matrix sample tests (matrix context) > and tiie examinees had not 
previously responded to the items (no exposure). 

Estlmatt^ 2. Data were collected during the administration of 
the loj^ [y^^^^ U asic Skills battery (normal context); and the 
oxaminees had, a 5; a result of Estimate 1^ previously responded to the 
items in t ht^ i r matrix tests (previous exposure). 

^^slimati^ ^. Data were collected during the administration of tlie 
^^^yi^^ l^a:; Lc S k 1 1 is battery (normal context); however, since the 

soroutl assij;nnirnt o{ n^atrix tests was used, tlie examinees had not 
previously rcsiK)n(UMl to the items (no exposure). 

Estimate 4. Data were collected by the administration of Set 2 
of the matrix sample tests (matrix context); and the examinees iiad » as 
a result oi'. the administration of the entire Iowa Tests of Basic Skills 
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TABLI- I 



CUNiU i l«»W!> or CONTl'XT AND KXPOSURbL KDR Tllb: FOUR 
SF.TS OF MATRIX SAMPLE ESTIMATKS 



Hst iin.U< 
I 



(.ontext 



|)»* r i viul t rtnn i\\e ndniinis- 
f r.iL i iMi i^r Set 1 of die 
matrix sample tests 



exposure 



Students had not: previously 
been exposed to any of Che 
items included in the 
matrix tests 



i pnst due t'roin data 
<-n)]piMed during the admin- 
i^-n.Uion o[ the entire ITUS ^ 
b;il t ^M'v s iinulat ing admin- 
ifUiMtion of Set 1 of the 
f'latrix sample tests 

P'M'ivtNi [lost hoc from data 
(dll^TlO'l during tlie admin- 
isirat inn of tlic entire ITBS 
11 1 t tM ' rv s f tnii ia t Ing admin- 
ist r.it ion of S(^t Z of tiie 
mat r i x sample tests 

Dorivod from administration 
i>f Sot 2 of the matrix sample 
tost s 



Students had previous 
exjx^'Sure to the Items 
saniple<l durinju; the 
administration of the 
niat.]'ix tests associated 
with Estimate 1 

Individual students had 
nv^L i^reviously been 
exposed to any of tliG 
(toins included in tlie 
Estimate 3 sample 



Students had previously 

been exposed to all 

items during the admin- 

i ;l ration of the entire 
LLli!^ battery 



low/i Test! 



d l^asir Skills 
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battery, previously responded to the items in their matrix tests 
(previous exposure), 

In addition to the four sets of multiple matrix samnlc estimate?;, 
ten sets of examinee sample estimates were computed, The examinee 
sample estimates were equivalent to the matrix sample estimates in that 
both were based on the same number of examinee-iten responses* The 
examinee sample estimates were computed by randomly sclectinj^ 21 fifth 
graders, 20 fourth fr^aders, and 18 sixth graders. The random selection 
was replicated 30 times since one replication v;as necessary to estimate 
each of the three Iowa Tests of Basic Skills subtests for each of the 10 
sets of estimates. The number of observations used in the matrix sample 
estimates and the examinee sample estimates are summarized in T?^ le Tl, 

Analysis 

The analysis consisted of comparing the multiple matrix estimates 
with the population parameters, the matrix context multiple matrix sample 
estimates with the post hoc matrix estimates, the previous exposure matrix 
sample estimates with the no previous exposure estimates, and the a priori 
multiple matrix sample estimates (Estimate 1) with the examinee samnle 
estimate. 

All estimates, whether matrix or examinee sampling estimates, were 
at one point or other compared with the counterpart population para- 
meters. For the comparison to be valid, the population parameters must 
be valid. If the prior administration of the matrix samnle tests biased 
tlic population parameters, then an adjustment would have to be made to 
compensate for the bias. 

The most lorical effect of the prior administration of the matrix 
tests was hi^^her scores on the subseouent testings. Such an 

8 



o 
:t: 

a ri 









LO 














TS 


S 












CO 








U] 


H 






UJ 








HH 


:-> 
to 


UJ 






to 








Tj 












X 


CO 


ui 








uu 






a 




H 


ui 

H 


IS 










P 






CO 


'J 



CO UmI 

ol - 

O CO 

< CO 

::> lu 

ft: H 

CO -t; 

.r) :^ 

o o 



UJ 



u o 
u H 

o 



to 
d 
o 



o 





O 












u 


r3 










.n 


u 




f1 


aj 






7) 






X) 






O 






(/) 






d 






o 


u 






r: 




U 


'1J 


O 




CJ 


H 










0) 






PL. 


O 


(/) 












O 




to 






d 




o 


o 






•t \ 










1) 






.n 


*> 


OJ 


f: 


>-( 


CO 






















O 












a 




O 


o 






•r-l 






U 




XI 




u 


.n 






f_: 


u 


to 




«D 






(n 






a:> 






O 




U 




\s\ 


U 




d 


x» 




0 


r". 










u 










o 


I 








ft! 




QJ 


iJ 




to 


O 




XI 


f-- 




O 



ON 


ON 


OA 


CO 


00 


00 


m 


m 


in 


























r H 


rH 











O 

CM 



II 

5S 



00 
II 



oo 


00 




O 


O 


c:> 


oo 


ac? 


O 


a\ 




in 








J 


r 4 












CO 


CO 


no 


CO 


00 















r ^ 






















v£) 








.-1 





















in 


ro 


in 


in 


00 


CO 


oo 


<r 


in 


in 




m 












00 


00 




CO 


00 


00 



00 


00 




ro 




m 


in 


in 


O 


00 


00 




in 


in 


m 


m 


CO 


CM 












CO 


CO 


00 


00 







<r 






or} 






m 






vO 








f-A 




o 






v-t 








o 


O 


0^ 


<r 






m 


in 




in 













>. 












( » 








fj 








u 






m 








>.o 


























i 








i-J 






OJ 


.-^ 


d 


03 


0) 




d 


(1 


u 




r: 


03 




(/> 








P 








f) 


-a 




•r K 


M 


CO 


oj 


r5 


.O 


r-H 


a> 


r3 




r -1 










OJ 


m 


tj 


Vj 




rH 


x: 


u 






.d 


^< 






x: 




.n 


O 


u 






O 


tj 


a' 


u 


a 


CJ 


0) 


u 








o 


CX 






o 


a. 






o 


a. 


>1 








> 


CO 














CO 



effect shoulJ be no.^t noticeable in matrix Estimate 2, which v;as 
based on the snnc itcrKs tliat were administered in the a priori 
matrix sannlin':^, Tlir^ro were no sipnificant differences between 
matrix finnnlinr Hr.t irate 2 and either Hstinato 1 or listimate 3, In 
^nct, tiic estinatcs in set 2 tended to be sliphtlv lower than the 
estinntos ir. tho other 5ets. Therefore, no adiustncnt in the popu- 
lation pararcter \:a«*> considered to be necessary, Tlie matrix sample 
estimates of the ncaijs and variances are found in Tables III anr^ IV, 

Analysis o^ Context and Exposure Tjffccts 

A 2 X 2 X ."S multivariate analysis of vaiiance design was 
utilized to test for tlic existence of a context and/or exposure 
effect. Tho dependent variables were the individual matrix test 
estimates ^or each of the three subtests. Vocabulary, Snellin^r and 
Matlicmatics . Two separate analyses were run, In t])c first, the mean 
scores were used as criterion variables, while in the second, the 
variance scores wore used as criterion variables. Tests of sip^ni- 
ficancc were computed for three main effects, context, exposure and 
grade level, and for the followin^r interactions: context-exposure, 
context-eracic, oxposurc-frrade, and context-exposurc-f^rade. Signifi- 
cant F ratios were found for contejtt effect utilizinn variances as the 
criterion fF statistic of 13,76 for 3, 13 df, p of less than 0,01) 
and for oxposM-rc effect utilizing? the means as the criterion (P statis- 
tics of 17.73 for 3, 13 df, p of less than 0,01), None of the inter- 
actions were sirnificant at the ,05 level. The Summary Tables for the 
tests of main effects are presented in Table V (A) . 
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TABLI-: III 



MULTIPLE MATRIX SAMPLE ESTIMATES OF THE MEAN 



ITBS* 
Subtest 


Population 
Mean 


Estimate 
1 


Estimate 
2 


Estimate 
3 


Estimate 
4 


VJ L CXKIKZ, H 














V W ^ CL \J \JL A.O L y 




21.177 




22 046 


71 778** 








18.798 




16 97 S 


1ft ftO?*** 

X O . \J\J i 


17 791** 






17.935 


i. O . ^ H 


J. o . ^ > w 


1 ft '^ft7*** 


1 Q SQ7 


f 
















25.202 


. \J sj 


?6 fi6Q 




27. 139 






19.807 






JO 1 1 7** 


IQ 614**5 


Mathematics 




19.933 


19.866*** 


20.067** 


18.908 


21.479 


Grade 6 














Vocabulary 




25.257 


24.625** 


26 1 6R** 

4L U « J. U U 


25.617** 


26.278** 


Spelling 




20.046 


20.714** 


18.290 


20.493** 


19.392** 


Mat hcrna t ic s 




19.450 


18.493 


20.579 


19.199** 


21.620 


* 

Iowa Tests 




Basic Skills 








Closer to the mean than five of the 
examinee sample estimates 


ten randomly 


drawn equivalent 


*** 

Clf)SGr to tlic mean than 
sample estimates 


all ten randomly dravm e 


quivalent 


examinee 
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TABLE IV 



MULTIPLE MATRIX SAMPLE ESTIMATES OF THE VARIANCE 



ITBS* 
Subtest 


Population 
Variance 


Estimate 
1 


Estimate 
2 


Estimate 
3 


Estimate 
4 


Grade 4 


















Vocabulary 


53.074 


JL . 


7n/i 


5-4.103*** 


4 J . 


/ o / 




/ J. 7"'' 


Spelling 


59.740 


. 


1 R/i 


'45. 86/4 


/ 0 . 


Q 7 7 


An 




Mathematics 


34.825 


JU . 




33.552** 




A 7 1 


J JL . 


717** 


Grade 5 


















Vocabulary 


59.739 


59. 


238*** 


52.1/48** 


61. 


556*** 


72. 


999 


Spelling 


66.784 


55. 


599*^^ 


53.08' ' • 


56. 


370** 


46. 


108 


Mathematics 


37.029 


39. 


252*** 


43.1-48** 


39. 


,566*** 


34. 


665**' 


Grade 6 


















Vocabulary 


66.711 


65. 


279** 


59./417** 


62. 


666** 


89. 


471** 


Spelling 


62.896 


56. 


A93** 


7/4.5/4/4** 


66. 


664** 


37. 


712 


Mathematics 


44.879 


60. 


923 


A8.153*** 


38. 


646** 


57. 


455 



leva Tests of Basic Skills 

Closer to the mean than five of the ten randomly drawn equivalent 
examinee sample estimates 



Closer to the moan than all ten randomly drawn equivalent examinee 
sample estimates 
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■ . . \ - 'ivjim: v (AJ. . ; . . v . . • 

SUMMAKY TAUKKS I'DH TKSTG Oi' IIAI M i:i'Kia;Tff 

?U)p:ia)'.v «aiwj-: for iiYJ'i)Tiii::;i;; i; comtivXT kki'kct on 

TU1-: liiSTJMATICJi OK TlIK MI-lAN'^ . 



Cun'piil.nJ. l<in V(>c:il)U Ijry wlpi' I. J i M;iL lu'inaLics 



UypoLhi'ii.i }' Wv'.wi 


ri'iuarutl l.?.f)/iO 


l^i.OUOA 


B.0526 


Univariate , K 


.5^03 


•6.51133 


2.7/i58 






■ .C22J 


.3183 


K- r.'it in f <ir i:m 


!l Iv.irlaLrj H;sL- of conl exL 
i t rccioiti, p leyr; tlum .1538. 


cf fccL - 2.0703 wicli 3 and 


SUllMAP.V ' 


TAr.l.l-: FOR l!YrOTlI!-:SI.S 2; CONTi:XT F.FFl-XT ON ' 

^•\\y. i-STiMATi-s or Tiijc VMa/m:\':^ 






Vocabulary 


Spcl.l J.nj; 


Ma rlicina t:.i cs 


Hypo [1 IP si ;'t Mcui 


Sfpiarcd 2^i2.5. 7G90 


7/415.9271 


352.3123 


Un.i.vr.u'i nl c F 


5.9923 


l/i./i603 


.3603 




.0272 


.0018 


.5573 


r i r> ft) r mn 
13 (Jc;;,j:c.(;:'. uC 


1 1 i var f ace t.cal: of contuxL 
liciHlom, p J.esiJ ili.')n »0003. 


cfCr'cL = 13.7555 


v;l:.li 3 aiul 


,SU>IMAiVV 


TAUl.i-. FOR 11Y1'0T11I::SJJS 3; . FXrOSMKi: I'FFl' CT ON 
TlIK I-STIMATKS OF 'niF MiiAH-v 




Coinpal..'a.iou 


Vocalm] ai*y 


Spel linjv 


MaUliomaiicG 


llypoLlio.s:ln Mc^-in 


Stpiarctl 35. 2272 


56.0A10 


37.7817 




25./J692 


28.yJ80 


16./.561 




.0002 


.OOOJ 


.001,1 


J'-rai 1 o Trir nut 
13 (]L;|^',i:ei'::; of 


ItivaM'aLc lent; of UApoi-^urc 
ficcflom, p less i:liaa .0001. 


. Ciiiv.vl -^17.7319 wA.tU 3 niul 


SUMMAIvY 


TAi^.u: i-ou !iyi*otiik:;i:s liXi'nsiiKi: ki'fi-x'j' on 

'\'\\V, KSTJ.MA'IF.S OF TDK VAKIAUCh* 




Cotiipitl a t J fii 


V(ii:aluila ry 


Spt? 1 t hiy, 


M.iLiu-m.'iLicii 




S'.|uav<:a C8A.5I30 


. (»'n .2G79 


3.5178 


Uii i.var iai f F 


. 6222 


: "J .0751 


.0071 


)' l.c^.ss Than 




■ .3163 


,..93m2 



F-rat in 1 im- m-i h i i a Lo I of i^:-;]M>;atri^ rrfocl. .6fj,''5 with 3 ar.«i 
'"" Q 13 ol I ifrdniii , p 3 iT.lJ rlu'Ui .5897. 



TABLE V 

DEVIATIONS OF MULTIPLE MATRIX SAMPLE ESTIMATES OF THE MEAN 
FROM ACTUAL POPULATION MEAN 



Subtos t 



Es t imate 
1 



Estimate 
2 



Estimate 

3 



Estiraaf 
4 



urade 4 

Vocabulary - .582 

Spelling .A91 

Mathematics .307 

Grade 5 

Vocabulary - .350 

Spelling 1.037 

Mathematics - .067 

Grade 6 

Vocabulary - .632 

Spelling .668 

Mathematics - .957 

Sum of 

deviations - .085 

Sum of Absolute 
Values of 

Deviations 5.091 



.869 
-1.823 
.355 

1.667 
-2.1A3 
.134 

.911 

-1.756 
1.129 

- .657 
10. 787 



.101 
.009 
.452 

.175 
.310 
■1.025 

.360 
.447 
■ .251 

.578 
3.130 



1.022 
-1.007 
1.662 

1.937 

- .193 

1. 546 

1.021 

- .654 
2.170 

7.504 
11. 212 



Iowa Tests of Basic Skills 
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TABLE VI 



DEVIATIONS OF MULTIPLE MrVTRIX SAMPLE ESTIMATES OF THE VARIANCE 

FROM ACTUAL POPULATION VARIANCE 



ITBS* 
Sub test 


Estimate 
1 


Estimate 
2 


Estimate 
3 


Estimat( 
4 


Grade 4 










Vocabulary 


- .370 


1.029 


- 7.307 


6.645 


Spelling 


-39.556 


-13.876 


17.232 


-19.258 


Mathematics 


- 4.428 


- 1.273 


- 4.354 


- 3.113 


Grade 5 










Vocabulary 


- .501 


- 7.591 


1.817 


13.260 


Spelling 


-11.185 


-13.700 


-10.414 


-20.676 


Mathematics 


o o o o 
Z . ZZ J 


0 . 119 


o Cot 

z . 53 / 




Grade 6 










Vocabulary 


- 1.432 


- /.Z94 


- 4.045 


ZZ . /oU 


Spelling 


- 6.403 


11.648 


3.768 


-25.184 


Mathematics 


16.044 


3.274 


- 6.233 


12.576 


Sum of 

Deviations 


-45.608 


-21.664 


- 6.999 


-15.354 


Sum of Absolute 
Values of 
Deviat ions 


82.142 


65.804 


57.707 


125.836 


Iowa Tests of 


Basic Skills 
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In addition to the multivarinte analysis of variance, deviation 
matrices were computed for both the estimates the mean and the vari- 
ance by subtractinr the appropriate population parameter from each of 
the nine estimates (three subtests for each of three prade levels) 
for each of the four sets of matrix sampling estimates, Two summary 
indices were computed for each deviation matrix, Tfie first, the sum 
of tlie deviations, was utilized as a measure of systematic bias, 
Estimates tliat were svs|:ematically too high would result in a larpe 
positive sum of the deviations, and estimates that were systematically 
too low would result in a larpe negative sum of the deviations. The 
second index, sum of the absolute values of the deviations, was an 
estimate of precision or variation. A relatively lar^e sum of the 
absolute values of the deviations would indicate that the estimates 
varied considerably, while a rclativelv small sum would indicate that 
the estimates were relativelv consistent. 

The deviation matrices for the mean can be found in Table V (B) 
and the deviation matrix for variance in Table VI. An analysis of the 
deviation scores indicated that with tl^e possible exception of Estimate 
4, the sum of tlie deviation of the mean tended to sum to zero, i.e. 
there were not systematic differences. The multiple matrix estimates 
of the variance tended to be too low; however, the sum of the deviations 
again approached zero with the exception of Estimate 1, The large 
nefrative sum of deviations for Estimate 1 appears to be an artifact of 
a bizzare estimate for spelling at the fourth eradc level. Multiple 
Matrix Sample Estimate 3, normal context-no previous exposure, was over- 
all the most accurate set of estimates of both the means and variances, 
Estimate 4 tended to be the v;orst estimate. 



ERIC 
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On the basis of the analysis, the ^ollowinp conclusions were 

made: 

1. The adninistration of the first set of natrix tests prior 
to the administration of the Iowa Tests of Basic Skills battery did 
not affect the examinee performance on the battery. 

2. No evidence was found for the existence of a context effect 
in the multiple matrix sample estimates of the mean. 

3. Chan^ye in item context si^fnificantly affected the estimates 
of tlie variance. The multiple matrix sample estimates of the variance 
computed from data collected by the actual administration of matrix 
tests showed greater variation on the deviation matrices than did the 
estimates computed from data collected durinr^ the administration of 
the entire battery. No evidence was found to indicate that tlie es- 
timates of the variance were systematically larj^er or smaller than 
would have been expected. 

4. Recent previous exposure to the items bcin^ samnled 
significantly affected the estimates of the mean. The estimates of 
the means computed from data that represented tlie examinees' second 
response to items within a week's time varied more than estimates com- 
puted from data that represented the examinees* ^irst response to the 
items. A^ain, no evidence was found that the Estimates of the mean 
were systematically lar^^er or smaller than would have been expected. 

5. Recent previous exposure to items in the samnled tests did 
not sijrnificantly affect tlie estimates of the variance. Both the means 
of the estimates and the variation al)OUt them were consistent for the 
four multiple matrix sample estimates. 
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Connnrison of natrix sannlinf^ estimates with examinee 
sampling estimates . Multiple Matrix Sample listimate 1 was the only 
matrix sample estimate used in this analysis. Estimate 1 approximated 
the way matrix samnlinji procedures would be used in an applied situa- 
tion . 

Deviation matrices and two summary statistics v;ere computed 
for each of the ten sets of examinee sample estimates. The summary 
indices were used to identify tlie examinee sample estimates tliat when 
compared v;ith Multiple Matrix Sample Estimate 1 would result in a 
conservative estimate of the precision of the matrix sample estimate. 
The deviation matrices for the means and variances are found in Tables 
yjl and VIII and the two sets of summary statistics are found in 
Tables IX and X. 

The set of examinee sample estimates that most accurately 
estimated the means anc: the set that most accuratelv estimated the 
variances were identified. A paired data t test was then used to 
compare the "most accurate" examinee sampling estimate with Multiple 
Matrix Sample Estimate 1. 

The sum of the absolute values of tlie deviations for Multiple 
Matrix Sample Estimate (estimates of the means) was smaller than the 
sums of the absolute values of the deviations of all ten sets of the 
examinee sample estimates of the means. The paired data t test between 
Estimate 1 and the most accurate set of examinee sample estimates was 
sij;nificant in a direction favoring the multiple matrix sample estimates. 
Therefore, the multiple natrix sample estimates of the means were 
concluded to be significantly better than examinee sample estimates of 
the means. 
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TABLE VII 



COMPARISON OF DKVIAHONS FROM THE POPUIjMION MEAN 
OF ESTIMATES OF THE MEAN OF MiMRIX SAMPLE 
ESTIMATE i AND TEN EQUIVALENT RANDOMLY 



DRAWN 


EXAMINEE SAMPLING 


ESTIMATES 




Estimate . 


Vocabulary 


Spelling 


Mathematics 


Grade 6 








Matrix Sample 


- .582 


.491 


.307 


Examinee Sample J 


.442 


-1.846 


- .649 


Examinee Snnjple 2 


.005 


- .131 


-1.413 


Examinee Sample 3 


- .558 


.^79 


- .649 


Examinee Sample 4 


.109 


-2.988 


- .887 


Examinee Sample 5 


- .177 


.964 


1.113 


Examinee Sample 6 


.680 


.916 


-1.268 


Examinee Sample 7 


1.442 


1.440 


.970 


Examinee Sample 8 


1.537 


-1.322 


-1.316 


Examinee Sample 9 


-3.225 


1.773 


- .887 


Examinee Sample 10 


.109 


1.392 


1.446 


Grade 5 








Matrix Sample 


- .350 


1.037 


- .067 


Examinee Sample 1 


-1.002 


3.143 


- .333 


Examinee Sample 2 


-1.202 


1.082 


-1.743 


Examinee Sample 3 


.348 


-2.507 


.96 7 


Examinee Sample A 


-1.552 


- .357 


.767 


Examinee Sample 5 


-1.452 


- .657 


1.317 


Examinee Sample 6 


1.648 


2.343 


.217 


Examinee Sanjple 7 


- .102 


- .207 


- .083 


Examinee Sample 8 


-1.202 


-1.157 


- .283 


Examinee Sample 9 


1.148 


.743 


3.417 


Examinee Sample 10 


-1.702 


1.443 


.167 


Grade 6 








Matrix Sample 


- .632 


.668 


- .957 


Examinee Sample 1 


1.632 


l.OU) 


.717 


Examinee Sample 2 


-1.728 


- .713 


.500 


Examinee Sample 3 


2.743 


- .102 


- .728 


Examinee Sample 4 


-3.757 


-2. 157 


- .172 


Examinee Sample 5 


.632 


2.954 


- .117 


Examinee Saiiiple 6 


- .924 


- .435 


-1.950 


Examinee Sample 7 


-2.035 


.343 


- .894 


Examinee Sample 8 


- .035 


-1.602 


.161 


Examinee Sample 9 


- .035 


-1 .879 


-2.561 


Examinee Sample 10 


- .979 


-2.379 


2.828 



ERIC 
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TABLE VIII 



COMPARISON OF DEVIATIONS FROM THE rOPUlATION VARIANCl:: 
OF FSTIMiVTKS OF THE VARIANCE OF mTRIX SiWLE 
ESTIMATE 1 AND TEN EQUIVALENT lUNDOMLY 
DRAWN EXAMINEE SAMPLING ESTIMATES 



Estimate Vocabulary Spelling Mathematics 



Grade A 



Matrix Sample 




- . J / U 


-39. 


556 




Examinee Samj>Ie 


1 


o . 274 


28. 


108 


o /I Q Q 

J . OoV 


Examinee Sample 




- 3.490 


1. 


498 




Examinee Sample 


J 


- 5 . 0 2 (1 


- 7. 


392 


f C Q O 
D • Doy 


Examinee Sample 


4 


15.0'tO 


- 1. 


778 


- 5.377 


Examinee Sample 


5 


5.526 


- 8. 


049 


14.523 


Examinee Sample 


6 


-26.245 




,826 


- 8.492 


Examinee Sample 


7 


- 7.726 


-10. 


,649 


- .134 


Examinee Sample 


8 


-13.160 




,622 


-16.477 


Examinee Sample 


9 


-24.426 


3. 


,217 


23.523 


Examinee Sample 


10 


-14.960 


-28. 


,678 


- 4.277 


Grade 5 












Matrix Sample 




.501 


-11, 


.185 


1 O 0 1 


bxaminee bampie 


L 


14 . 114 




,845 




Examinee Sample 


'y 


- 7. job 




,850 


-Id . ud / 


Examinee Sample 


T 
J 


y . 15 J 


18.385 




Examinee Sample 


4 


15.553 


25.161 


- 5.334 


Examinee Sanjple 


5 


20.143 


11 , 


.24 5 


- 6.200 


Examinee Sample 


() 


3. 343 


-11, 


.59 7 


- 9.526 


Examinee Sample 


7 


- 2.592 


15. 


, 363 


- 9.632 


Examinee Sample 


8 


- 7.844 


22, 


.613 


-24. 158 


Examinee Sample 


9 


- 2.657 


-16, 


.102 


- 5.421 


Examinee Sample 


10 


13.261 


37. 


,519 


10.013 


Grade 6 












Matrix Sample 




- 1.432 


- 6, 


.403 


16.044 


Examinee Sample 


1 


3. 394 


-14, 


.487 


12.797 


Examinee Sample 


2 


24 . 304 


- 9, 


.014 


- 6.197 


Examinee Sample 


3 


.818 


-22 


.017 


5.098 


Examinee Sample 


4 


-32.917 


29 


.562 


-13.961 


Examinee Sample 


5 


-12.724 


10.045 


-10.879 


Examinee Sample 


6 


27.642 




.885 


12.562 


Examinee Sample 


7 


-22.646 


65, 


. 709 


- 6.382 


Examinee Sample 


8 


43.119 


- 4 


.517 


- 6.039 


Examinc?e Sample 


9 


3.707 


-22 


.043 


- 7.951 


Examinee Sample 


10 


31.619 


-17 


.131 


14.510 




TABLE IX 



COMrARlSON OF THF: SirM OF DEVIATIONS AND THE SUM OF llUi 
ABSOLUTE VALUES OF DEVIATIONS FROM THE POPULATION MEAN 
OF ESTIMATES OF THE MEAN OF MATRIX SAMPLE 
ESTTMy\TE 1 AND TEN EQUIVALENT RANDOMLY 
DRAWN EXAMINEE SAMPLING ESTIMATES 



Es Cim£iCG 




Slim of 
Dev ia t ions 


Sum of Absolute 
Values of Deviations 


Matrix Sample 




- .085 


5.091 


Examinee 


Sample 


1 


3.114 


10.774 


Examinee 


Sample 


2 


- 5.343 


8.517 


Examinee 


Sample 


3 


- .307 


8.7S1 


Examinee 


Sample 




-10.994 


12.746 


Examinee 


Sample 


5 


4.577 


9.383 


Examinee 


Sample 


6 


1.227 


10.381 


Examinee 


Sample 


7 


.874 


7.516 


Examinee 


Sample 


8 


- 5.219 


8.615 


Examinee 


Sample 


9 


- 1.506 


15.668 


Examinee 


Sampl e 


10 


2.3/5 


12.445 



21 



TABLE 



COMPARISON OF THE SUM OF DEVIATIONS AND TllF SITM OF THE 
ABSOLUTE VALUES OF DEVIATIONS FROM THE POPULATION 
VARIANCE OF ESTIMATES OF THE VARIANCE OF MATRIX 
SAMPLE ESTIMATE 1 AND TEN EQUIVALENT RANDOMLY 
DRAWN EXAMINEE SAMPLING ESTIMATES 



Sum of Sum of Absolute 

Estimate Deviations Values of Deviations 



Matrix Sample 




-45.608 


82 . 142 


Examinee 


Sample 


1 


61.958 


90.932 


Examinee 


Sample 


2 


-11.430 


72.878 


Examinee 


Sample 


3 


9.726 


78.596 


Examinee 


Sample 




25.949 


144.683 


Examinee 


Sample 


5 


23.620 


99.334 


Examinee 


Sample 


6 


-12.254 


101.118 


Examinee 


Sample 


7 


21.311 


140.833 


Examinee 


Sample 


8 


- 5.841 


138.549 


Examinee 


Sample 


9 


-48.153 


109.037 


Examinee 


Sample 


10 


41.876 


171.968 
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The sum of the absolute values of the deviations for Multiple 
Matrix Sample nstimate 1 (estimates of the variances) was smaller 
than the sums of the absolute values of the deviations of eipht of 
the ten' sets o^ examinee sample estimates of the variances, The 
paired data £ test between Estimate 1 and tlie most accurate set of 
examinee sample estimates was not si^rnificant. Therefore, it was con- 
cluded that the multiple matrix sample estimates of the variances 
were as accurate as comparable examinee sample estimates of the 
variances . 

Conclusions 

This study once again demonstrated that multiple matrix 
sampling is an effective procedure for collecting data on the per- 
formance of proups. An a priori set of nine multiple matrix sample 
estimates, one for each of three subtests of the Iowa Tests of Basic 
Skills (Vocabulary, Spelling and Mathematics Concepts) for each of the 
three grade levels (fourth, fifth and sixth), was si.enif icantly more 
precise than ten similar sets of examine sampling estimates. No signi- 
ficant differences were found between the multiple matrix sample es- 
timates and examinee sample estimates of the variances. 

The findings regarding the effect of the changes in item 
context necessitated by matrix sample procedures and the effect of 
previous exposure to items on the matrix estimates were encouraging. 
The change in item context did not significantly affect the matrix 
sample estimates of the mean, but it did affect the estimates of the 
variance. Conversely, previous exposure to items affected the matrix 




sample cstinates of the mean but not t)^»e estimates of the variance. 
Botli tlie context and exposiVxe effect involved an increase in the 
variation of the estimates and, there-f^ore, a decrease in precision. 
Neither effect seemed to cause the estimates to he either system- 
atically too hip,h or too low. The loss in precision could l^e compen- 
sated for by increasing the number of observations. A systematic 
bias would have been nuch more vexinjr. The results, as encouraginr 
as they were, should he interpreted cautious Iv. This study needs to 
be replicated in other settin^rs usinrr other instruments. 
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